
Calhoun 

iniQiuiic^iul Ar{hiv« of tilt Mil vdl Poii^roduiit School 


Calhoun: The NPS Institutional Archive 
□Space Repository 



Theses and Dissertations 


1. Thesis and Dissertation Collection, all items 


2009-12 

Distributed and end-to-end testing. 


Le, BeEm V. 

Monterey, California. Naval Postgraduate School 


http://hdl.handle.net/10945/4506 


Downloaded from NPS Archive: Calhoun 



DUDLEY 

KNOX 

LIBRARY 


htt p://w ww. n ps. e du/l ib ra ry 


Caflwuo is the Naval Postgraduate School's public access digital repository for 
research mate rials and institutiional putilicatiiaos created by the NPS community. 
Calhoun is named for Professor of Mathematics Guy K. Caftiouo, NPS's first 
appointed — and putJlished — schoteily author. 

Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 Univefsity Circle 
Monterey, California USA 93943 







NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY, CALIFORNIA 


THESIS 


DISTRIBUTED AND END-TO-END TESTING 

Thesis Advisor: 

by 

BeEm V. Le 

December 2009 

Thomas V. Huynh 

Second Reader: 


Raymond Barrera 


Approved for public release; distribution is unlimited 



THIS PAGE INTENTIONALLY LEET BLANK 



REPORT DOCUMENTATION PAGE 


Form Approved 0MB No. 0704-0188 

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, 
searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send 
comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to 
Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 
22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503. 

2. REPORT DATE 3. REPORT TYPE AND DATES COVERED 
December 2009 Master’s Thesis 

4_TITLE^ND_Sl]DTITLE^_^istributedandEnd^to^EndTestin^^^^^^^^^^_ 5. EUNDING NUMBERS 

6. AUTHOR(S) 

7. PEREORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PEREORMING ORGANIZATION 

Naval Postgraduate School REPORT NUMBER 

9. SPONSORING /MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING/MONITORING 
N/A AGENCY REPORT NUMBER 

II. SUPPUEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the official policy 
or position of the Department of Defense or the U.S. Government._ 


13. ABSTRACT (maximum 200 words) 

Many U.S. Navy systems were built on the fly and have encountered interoperability problems at sea, such as 
erroneous dual/multiple track designations, misidentification/track identity conflicts, report responsibility conflicts, 
friendly tracks displayed as unknown/pending, tracks dropped without operator action, different track identities of at 
different ships, etc. To identify and fix these interoperability problems, the Navy instituted the Distributed 
Engineering Plant (DEP) testing program, run by the Naval Sea Systems Command (NAVSEA), and the End-to-End 
(E2E) testing initiative, currently formed by the Space and Naval Warfare Systems Command (SPAWAR). Whereas 
the DEP involves many land-based laboratories across the U.S. connected via an Asynchronous Transfer Mode 
(ATM) network, E2E testing is carried out entirely at one laboratory—the E2E lab. The DEP testing program is faced 
with the problem of determining a cost-effective way of paying for testing—providing the participant DEP 
laboratories full-time funding or paying them on a per-test basis. A challenge faced by the E2E testing program is 
getting the E2E lab ready for testing. Two factors contributing to this challenge are uncertain availability of funding 
for building the E2E lab and the lack of a comprehensive plan to establish the E2E lab. Such a plan calls for a 
rigorous justification of the E2E lab needs and hence funding requirements. This thesis performs an in-depth 
examination and a qualitative analysis of the two testing programs and a quantitative comparative analysis of the DEP 
testing program’s paying options and, using goal programming, provides data in support of creating an E2E lab plan. 
The significance of this thesis is the use of analysis and mathematical programming to provide analytical data in 
supporting informed decision making in testing and evaluation of systems and/or systems of systems. 


NSN 7540-01 -280-5500 Standard Form 298 (Rev. 2-89) 

Prescribed by ANSI Std. 239-18 


14. SUBJECT TERMS 

Distributed, End-to-End, Combat Systems, C4I Systems, Pay-per-Test, Goal Programming (GP), 
Distributed Engineering Plant (DEP), E2E Lab, PMW, PEG C4I, NAVSEA, SPAWAR 

18. SECURITY 
CLASSIEICATION OE THIS 
PAGE 

Unclassified 


19. SECURITY 
CLASSIEICATION OE 
ABSTRACT 

Unclassified 


17. SECURITY 
CLASSIEICATION OE 
REPORT 

Unclassified 


15. NUMBER OE 
PAGES 

_81_ 

16. PRICE CODE 

20. LIMITATION OE 
ABSTRACT 


12b. DISTRIBUTION CODE 


12a. DISTRIBUTION / AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited 


1. AGENCY USE ONLY (Leave blank) 


1 




























THIS PAGE INTENTIONALLY LEET BLANK 


11 



Approved for public release; distribution is unlimited 


DISTRIBUTED AND END-TO-END TESTING 


BeEm V. Le 
Department of the Navy 
B.S., Bradley University, 1987 


Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF SCIENCE IN SYSTEMS ENGINEERING MANAGEMENT 

from the 


NAVAL POSTGRADUATE SCHOOL 
December 2009 


Author: 


BeEm V. Ee 


Approved by: Dr. Thomas V. Huynh 

Thesis Advisor 


Mr. Raymond Barrera 
Seeond Reader 


Dr. Cliff Whitcomb 

Chairman, Department of Systems Engineering 



THIS PAGE INTENTIONALLY LEET BLANK 


IV 



ABSTRACT 


Many U.S. Navy systems were built on the fly and have eneountered 
interoperability problems at sea, sueh as erroneous dual/multiple traek designations, 
misidentifieation/traek identity eonfliets, report responsibility eonfliets, friendly traeks 
displayed as unknown/pending, traeks dropped without operator aetion, different traek 
identities of at different ships, ete. To identify and fix these interoperability problems, 
the Navy instituted the Distributed Engineering Plant (DEP) testing program, run by the 
Naval Sea Systems Command (NAVSEA), and the End-to-End (E2E) testing initiative, 
eurrently formed by the Spaee and Naval Warfare Systems Command (SPAWAR). 
Whereas the DEP involves many land-based laboratories across the U.S. connected via an 
Asynchronous Transfer Mode (ATM) network, E2E testing is carried out entirely at one 
laboratory—the E2E lab. The DEP testing program is faced with the problem of 
determining a cost-effective way of paying for testing—providing the participant DEP 
laboratories full-time funding or paying them on a per-test basis. A challenge faced by 
the E2E testing program is getting the E2E lab ready for testing. Two factors 
contributing to this challenge are uncertain availability of funding for building the E2E 
lab and the lack of a comprehensive plan to establish the E2E lab. Such a plan calls for a 
rigorous justification of the E2E lab needs and hence funding requirements. This thesis 
performs an in-depth examination and a qualitative analysis of the two testing programs 
and a quantitative comparative analysis of the DEP testing program’s paying options and, 
using goal programming, provides data in support of creating an E2E lab plan. The 
significance of this thesis is the use of analysis and mathematical programming to 
provide analytical data in supporting informed decision making in testing and evaluation 
of systems and/or systems of systems. 
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EXECUTIVE SUMMARY 


Distributed and End-to-End Testing in the U.S. Navy 

Dealing with test and evaluation of eombat and Command, Control, 
Communieations, Computers and Intelligenee (C4I) systems in the U.S. Navy, this thesis 
foeuses speeifieally on two separate Navy testing programs managed by two different 
eommands: the Distributed Engineering Plant (DEP) testing program, run by the Naval 
Sea Systems Command (NAVSEA), and the End-to-End (E2E) testing program, 
eurrently formed by the Spaee and Naval Warfare Systems Command (SPAWAR). 

During past deployments and Battle Group exereises. Battle Group systems 
eneountered interoperability problems such as erroneous dual/multiple track designations, 
misidentification/track identity conflicts, report responsibility conflicts, friendly tracks 
displayed as unknown/pending, tracks dropped without operator action, different track 
identities of different ships, etc. As new technologies and commercial-off-the-shelf 
(COTS) products, used in the development of the systems, changed rapidly (items usually 
becoming obsolete in a couple of years), and lacked backward compatibility, the systems 
failed to work together, thereby resulting in those interoperability problems. When these 
interoperability problems occurred at sea, fixing them was costly because of the high cost 
incurred in bringing subject matter experts from land to ships to correct the problems. In 
an effort to prevent these costly problems, the DEP testing program was formed in June 
1998 to support the evaluation of the interoperability of Battle Group systems and also to 
aid in design decision making early in the systems acquisition process. 

The DEP consists of shore-based combat system sites such as the Naval Surface 
Warfare Centers (NSWC) in Dahlgren, Dam Neck, Wallops Island, and San Diego; the 
SPAWAR Systems Center-Pacific (SSC-PAC); and the Naval Air Systems Command 
(NAVAIR) Paxtuxent/China Eake. These combat system sites, connected via an 
Asynchronous Transfer Mode (ATM) network provided by Defense Information Systems 
Agency (DISA) and using Defense Information Systems Network-Eeading Edge Services 
(DISN-EES), replicate, to the maximum extent possible, hardware, computer programs, 
connectivity, and the environment of the ship and aircraft combat systems. To replicate a 

xvii 



Battle Group, the DEP would include the combat system sites representing the members 
of the Battle Group. A combat system would be connected, not to all sites, but only to 
the sites that could serve as test beds for the required platform combat systems. 

The purpose of the E2E testing is to evaluate integrated capabilities of shipboard 
C4I systems for interoperability. The shipboard C4I systems, which provide enhanced, 
integrated C4I capabilities through integrated communication and information technology 
systems that would deliver end-to-end connectivity to aid in achieving decision 
superiority, need be tested before their delivery to the warships, with the hope to prevent 
interoperability failures from occurring while the warships are at sea. In addition, not 
only does the E2E testing support the certification, but it also supports the developmental 
testing of multiple programs. Concentrated mainly on C4I systems, the E2E testing 
program emphasizes the area of services, such as Domain Name Server/Email/Web; 
communication systems, such as Consolidated Afloat Networks and Enterprise 
Services/Advanced Digital Network System/Teleport/Network Operation Center; 
Information Assurance, such as Gold DislCVirtual Private Network/Cryptos; and 
networks, such as Satellite Communications & Eine of Sight & Pierside. In the future, 
the E2E testing may be expanded to include systems outside of the C4I arena. 

In the E2E testing concept, formulated in 2007, a single E2E systems engineering 
laboratory, which is yet to be built, replicates and tests these C4I systems. The E2E lab is 
to house as much C4I hardware as possible in order to support the E2E testing. Eimited 
in funding, the E2E testing program plans to employ test engineers from the Program 
Manager Warfare (PMW) organizations and to have a lab manager, a lab technician, 
network engineers, and systems administrators to perform day-to-day lab activities. 

A challenge faced by the E2E testing program is building and getting the E2E lab 
ready for testing. Two factors contributing to this challenge are uncertain availability of 
funding for building the E2E lab and the lack of a comprehensive plan to establish the 
E2E lab. Such a plan calls for a rigorous justification of the E2E lab needs and hence 
funding requirements. Analytical support is needed to provide rigorously obtained data 
to aid in the formulation of such a plan. 



With a focus on the issue of cost-effeetive implementation of the DEP and E2E 
testing programs, the researeh performed in this thesis involves (1) conducting a review 
of the past and current test results of the distributed, E2E programs and their archived 
doeuments, related distributed and E2E testing studies, and other pertinent related 
material such as test reports, test plans, and test proeedures; (2) developing interview 
questionnaires directed at managers of the DEP and E2E testing programs; (3) eonducting 
interviews with the DEP and E2E testing program managers, test directors, functional 
leads, and process literature and interview results; and (4) performing qualitative and 
quantitative analysis and goal programming to aid in determining the optimal costs of 
carrying out these testing methods. 

The findings from this researeh follow. 

DEP Testing Program 

• The DEP program eurrently pays the DEP participant sites on a per-test basis, 
rather than on a retaining full-time basis. Perceived by the DEP program as a 
eost-effeetive paying concept, the pay-per-test method does not pay the DEP 
sites to retain engineers upon the completion of testing. When a test is 
eompleted, the DEP partieipant sites, whose funding is now depleted, lose the 
experieneed engineers. When a new test or retest is needed, newly hired 
engineers, who have little or no knowledge of the program, use a major 
portion of the allocated testing time to learn and set up the test. Also, when 
any of the DEP sites has trouble and consequently eannot join the remaining 
DEP sites for testing as planned, the latter have to wait until the troubled site 
is fixed. As a result, the alloeated testing time is not fully used for testing. 
The data analysis in this research indicates that roughly 38% of the alloeated 
testing time is lost at the beginning of a test. The time loss, resulting from the 
current pay-per-test praetice, remains a major issue the DEP program needs to 
resolve. 

• Another paying option for the DEP program to consider is paying the DEP 
participant sites a fixed amount of money to retain full-time engineers. This 



option is not necessarily cost-effective as the sites will constantly deplete 
funding, whether they are doing testing or staying idle. Using the testing cost 
data from FY01-FY09, during which the number of tests carried out ranges 
from three to eight per year, the comparative analysis performed in this 
research, evaluating the two options—pay-full-time or pay-per-test—indicates 
that conducting eight tests during FY09 with two engineers, using the pay-per- 
test method costs approximately $3.8M whereas using the full-time payment 
method costs about $2.7M. Furthermore, the analysis indicates that if the 
number of tests per year is not greater than six, the pay-per-test method is 
cheaper than the pay-full-time option and that if more than six tests per year 
are carried out, the full-time option is cheaper than the pay-per-test option. As 
six or less tests are conducted for only two out of the nine years (FYOl and 
FY05), the pay-full-time option would have collectively saved money during 
FY01-FY09. In this case, the core experienced engineers would have been 
likely retained and the lost time would have been reduced, if not eliminated. 
Thus, in the long run, if more than six tests are conducted, savings will be 
made with the pay-full-time option. Finally, if the number of tests is less than 
six per year, the DEP program has the flexibility in selecting either of the 
paying options. 

E2E Testing Program 

• Using testers from the PMWs, as currently planned by the E2E testing, is not a 
desirable approach, as the PMW testers would leave after completion of the 
testing and would also take with them their knowledge, which might not be 
available when needed for additional testing. This approach would possibly 
lead to the time loss problem faced by the DEP program; a lack of 
accountability as different PMWs from which testers come would claim 
effectiveness when, in actuality, history has shown otherwise; linger pointing 
in the case of failure; difficulty in timely coordination and flexibility of 
schedule in order to support the E2E testing program; and possibly not 
meeting the schedule. 
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• Rather being Program Objeetive Memorandum (POM) funded, the E2E 
testing program secures its funding from taxing the PMWs. As such funding 
is uncertain, the E2E testing program might not be sustainable in the future. 

• The first planned E2E testing event scheduled for December 2008 of the 
Eincoln Battle group did not happen because the E2E lab has not been built 
yet. Eunding uncertainty and the lack of a complete plan to establish the E2E 
lab contribute to the delay in building and getting the E2E lab ready for 
testing. 

• Toward the formulation of a comprehensive plan to build the E2E lab, the use 
of goal programming is demonstrated in this thesis. The demonstration 
reveals that, for the scenario treated in this research, the planned cost for 
building the E2E lab, which is $650,000, exceeds the optimal cost obtained 
with the goal programming approach, which is $647,000, by $3,000. Whereas 
money savings are realized, the targeted number of desks and chairs is short 
by one. This negligible shortage is acceptable. 

• Einally, goal programming can be used as a rigorous approach to determining 
the goals of the E2E lab in a timely fashion (hence, saving money) that can 
meet budgetary constraints. The E2E testing program can use this approach to 
justify the funding for the current year and future funding. The goal 
programming approach can also be used in support of planning for programs 
other than testing as well as for many multi-goal problems encountered in 
systems engineering. Results obtained with this goal programming approach 
are used in decision making, in assessing the feasibility of achieving the goals, 
and also in knowing how much determining funding is required to meet the 
goals. 

Recommendations 

• Eund the DEP sites on a pay-per-test basis if no more than six tests are 
performed in a year; otherwise, fund the DEP sites on a full-time basis. 

• Retain the core engineers to run DEP tests. 
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• Employ optimization techniques in general and goal programming in 
particular in planning for and, specifically, in formulating a plan for building 
the E2E lab. 

• Employ rigorous approaches — analytical and/or optimization techniques — 
from the beginning of testing programs to plan for and implement the testing 
programs. 

Areas for Further Research 

• Test and evaluation communities may extend the analysis and optimization 
techniques discussed in this thesis to determine the optimal methods of 
conducting tests with constraints on resources such as laboratories, personnel, 
schedules, etc. 

• The approach espoused in this thesis may serve as a foundation for additional 
research and studies of the joint distributed testing/coalition distributed 
testing. 
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I. 


INTRODUCTION 


A. BACKGROUND 

This thesis deals with some aspeets of test and evaluation of systems in the U.S. 
Navy. Speeifieally, this thesis foeuses on the distributed testing and end-to-end (E2E) 
testing programs. Briefly, whereas distributed testing involves many geographieally 
distributed sites (labs) that are intereonneeted to earry out a test, E2E testing is carried out 
entirely in one lab (at one site). By virtue of the purposes of these test programs, only 
combat systems and command, control, communications, computers, and intelligence 
(C4I) systems are subjects of testing. 

Interoperability among the Elect units in deployment fails as a result of erroneous 
dual/multiple track designations, misidentification/track identity conflicts, report 
responsibility conflicts, friendly tracks displayed as unknown/pending, tracks dropped 
without operator action, etc. (Monteith, 2001). In the absence of a test and evaluation 
program, fixing these interoperability failures adversely affected the Elect’s missions. In 
one instance, a Battle Group was deployed without two modern combatants, as the latter 
were tied to a pier in order to have interoperability problems fixed. In another instance, 
a great deal of time during the final six months prior to a Battle Group deployment was 
consumed, not by training, but by “debugging” of systems in the Battle Group. These 
unacceptable instances called for a systematic testing and evaluation approach to deal 
with the interoperability problems (DEP, 2005). 

To address combat systems interoperability problems across Battle Management 
Command, Control, Communications, Computers and Intelligence (BMC4I)/combat 
systems and to work with the Elect to fix the interoperability problems, the Naval Sea 
Systems Command (NAVSEA), in April 1998, formed the Combat System 
Interoperability Task Eorce, whose objectives were to examine the interoperability crisis 
and to provide recommendations and/or solutions for fixing the interoperability problems. 
To achieve these objectives, the Task Eorce determined the feasibility and cost of using a 
land-based distributed engineering plant (DEP) to support the design, development. 
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testing, and evaluation of Battle Group systems (DEP, 2005). In June 1998, the DEP was 
established, signifying the beginning of the Navy’s distributed testing. 

Distributed testing has, however, eneountered the diffieulty of getting all the labs 
together and having all resources available at one time to form the DEP. The DEP 
program currently pays the DEP participant sites on a per-test basis, rather than on a 
retaining full-time basis (Mann, 2004). Perceived by the DEP program as a cost-effective 
paying concept, the pay-per-test method does not pay the DEP sites to retain engineers 
upon the completion of testing. When a test is needed, inexperienced part-time engineers 
are hired to perform the test. This practice has led to testing inefficiency, namely, the 
loss of test time to setting up the labs by inexperienced part-time engineers, and need to 
be evaluated against the full-time paying option. One of the goals of this thesis research 
is to perform analyses to aid the DEP program manager (PM) in making the correct 
decision in paying the DEP labs for testing. Chapter II discusses in detail the DEP testing 
program and its problems/issues. 

The concept of E2E testing in the Navy began in 2007, when Office of the Chief 
of Naval Operations (OPNAV) tasked the Navy Program Executive Officer Command, 
Control, Communications, Computers, and Intelligence (PEO C4I) to provide enhanced, 
integrated C4I capabilities through integrated communication and information technology 
systems that would deliver end-to-end connectivity to aid in achieving decision 
superiority (Miller, 2008). These integrated communication and information technology 
systems often make use of commercial of the shelf (COTS) technologies. When these 
systems encounter problems at sea, fixing them can become costly because of the high 
cost incurred in bringing subject matter experts (SMEs) from land to ships to correct the 
problems at hand. Requested by PEO C4I to provide support in solving this cost 
problem, the Space and Naval Warfare (SPAWAR) Systems Center-Pacific (SSC-PAC) 
is building an E2E Systems Engineering (SE) lab (E2E SE, 2008), which will replicate 
and test multiple shipboards C4I systems before their delivery to the Elect. Building the 
E2E lab can be costly, and the utilization of the E2E lab, along with the use of personnel, 
is also an issue. The second goal of this thesis research is to analyze the current approach 
to building the E2E lab and to explore the use of goal programming as an optimization 
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tool to aid the E2E testing PM in optimally formulating a plan to build the E2E. Chapter 
III diseusses in detail the E2E testing effort and its problems/issues. 

The rest of this chapter discusses the purpose of the thesis (Section B), the 
research questions (Section C), the potential benefits of the thesis (Section D), the scope 
and the research methodology (Section E), and the organization of the remaining of the 
thesis (Section E). 

B. PURPOSE 

With a focus on the issue of cost-effective implementation of the DEP and E2E 
testing programs, the research performed in this thesis identifies the issues and problems 
encountered by the DEP and E2E testing programs; examines available information and 
data from the two programs; performs qualitative and quantitative analyses to provide 
data to aid the DEP testing program with its implementation; and explores and applies 
goal programming to support the formulation of a plan to build the E2E lab. 

C. RESEARCH QUESTIONS 

The primary questions are: 

1. How does the Navy employ distributed testing and E2E testing? 

2. Can optimization methods be used to aid in determining the minimal cost in 
implementing these testing methods while maximizing the use of available 
resources (personnel and lab hardware)? 

Answering these questions amounts to answering the following secondary 
questions: 

a. What are the differences between the distributed testing and E2E testing 
programs? 

b. What are the current methods used by the Navy to conduct distributed and 
E2E testing? 

c. How does the Navy employ these testing methods for testing of naval combat 
systems? 
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d. What analysis and optimization methods that can be used to aid in 
determining the minimal costs of carrying out these testing methods while 
maximizing the use of lab resources (including hardware and personnel)? 

e. What recommendations regarding the implementation of these testing 
methods can be provided to the Navy? 

D. RESEARCH BENEFIT 

This thesis demonstrates the analysis and the use of goal programming methods 
that can be used by PMs to effectively and efficiently perform their tasks of budgeting, 
scheduling of lab assets, and utilization of the labs. In addition, the Test and Evaluation 
community may extend the analysis and optimization techniques discussed in this thesis 
to determine the optimal methods of conducting tests with constraints on resources such 
as laboratories, personnel, schedules, etc. Furthermore, the approach espoused in this 
thesis may serve as a foundation for additional research and studies of the joint 
distributed testing/coalition distributed testing. 

E. SCOPE AND METHODOLOGY 

1. Scope 

With a focus on the issue of cost-effective implementation of the DEP and E2E 
testing programs, the research performed in this thesis identifies the issues pertaining to 
the two major problems encountered by the DEP and E2E testing programs; examines 
available information and data from the two programs; performs qualitative and 
quantitative analyses to provide data to aid the DEP program with its implementation; 
and explores and applies goal programming to support the formulation of a plan to build 
the E2E lab. 

2. Methodology 

The research methodology involves: 

a. Conducting a review of the past and current test results of the 
distributed, E2E programs and their archived documents, related 
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distributed and E2E testing studies, and other pertinent related material 
sueh as test report, test plans, and test proeedures; 

b. Developing interview questionnaires directed at managers of the DEP 
and E2E testing programs; 

c. Conducting interviews with the DEP and E2E testing PMs, test 
directors, functional leads, and process literature and interview results; 

d. Performing qualitative and quantitative analysis and goal 
programming to aid in determining the optimal costs of carrying out 
these testing methods; and 

e. Eormulating recommendations to the Navy regarding the 
implementation of these testing methods. 

F. THESIS ORGANIZATION 

The rest of the thesis is organized as follows. Chapter II discusses the DEP 
testing program. Chapter III discusses the E2E testing program. Chapter IV presents the 
analyses of both the DEP and E2E testing programs. Chapter V discusses the 
comparative analysis for aiding the DEP testing program in making paying decisions and 
the goal programming approach in supporting the E2E testing program. Chapter VI 
contains the conclusions and recommendations on future research. 
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II. DISTRIBUTED ENGINEERING PLANT PROGRAM 


A. INTRODUCTION 

This chapter provides the background of and the problems eneountered by the 
DEP program. As mentioned in Chapter I, the Navy DEP was formed to aid in solving 
interoperability problems such as communications between ship systems, common 
operational pictures between ships, ineorreet IDs, dual tracks, etc. that occurred during 
deployments and Battle Group exereises (Monteith, 2001). Interoperability problems 
encountered by combat ships have had their roots in the use of commereial-off-the-shelf 
(COTS) products and new technologies to develop systems. As technologies change 
rapidly (items usually becoming obsolete in a couple of years) and baekward 
eompatibility of some items has been a common phenomenon, the use of COTS and 
constant technological advances has ereated aequisition problems not only for the Navy, 
but also for the Department of Defense (DoD). Interoperability problems have become a 
challenge to the Navy. 

In Eebruary 1998, the Eleet reported concerns regarding interoperability failures 
among combat systems that were reeently installed in the Eleet units (USS John E. 
Kennedy and USS Wasp). Debugging of glitehes eonsumed a great deal of Eleet time 
during the final six months prior to Battle Group deployment (DEP, 2005). In March 
1998, the Chief of Naval Operations assigned the Naval Sea Systems Command 
(NAVSEA) the responsibility to address combat systems interoperability problems across 
BMC4I/combat systems and to eoordinate an effort with the Eleet to solve the 
interoperability problems. In April 1998, NAVSEA formed the Task Eorce on Combat 
System Interoperability to study the interoperability crisis and to provide 
recommendations for solutions. In June 1998, the Task Eoree affirmed the teehnieal 
feasibility of the establishment of a DEP to support the evaluation of the interoperability 
of battle force systems and to enable good design deeisions early in the aequisition 
proeess (DEP, 2005). 
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Following the Task Force Report, fourteen Navy organizations shown in Figure 1, 
representing the surface, air, subsurface, and C4I components across all Navy Systems 
Commands (SYSCOMs), formed the Battle Force Interoperability Navy Alliance (DEP, 
2005) to cooperatively support the interoperability task. The initial purpose of the Navy 
Alliance was to develop a proposal for the establishment and implementation of a Navy 
DEP. 
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Aeg 8 Cbm ba t S ysfa ns C ent er - Wal ops I si and, V A 
l^va IWarfare Aial ysfa Sta ton -Cb o na, 
hfava I Ohde s ea Warfare Cfanter v\p orf H 
hfava I Sur fa ce Warfare Cfan far/Pl-D -Oxnard, CA 
SPA R Sys fa ns C ent er- P act i c- S an Cfa go, C A 

hfava I Sur fa ce Warfare Cfan far/ Pl-D - D am N eck, V A 

SPA VA R Sys fa ns C ent er- AI ant b - C har fa sb n, S C 

hbva I Sur fa ce Warfare Cfan far/ Pl-D -S an Cfa go, 0\ 

Aeg 8 Tran hg and fta cii ess Cfan far -Dahlgren, V A 
hfava I %se arch Laborat ory - Al hgton , V A 
JHU A ppl fad P hysi cs Labo rab y - Laurel, M D 
hfava I A rWarf areC enter/AD -Rituxe ntRi/er, l\D 
hfava I A rWarfareC enterWD -Chi na L ake, C A 


Eigure 1. NAVY DEP Alliance (DEP, 2005) 


The DEP consists of shore-based combat system sites such as the Naval Surface 
Warfare Centers (NSWC) in Dahlgren, Dam Neck, Wallops Island, and San Diego; the 
SPAWAR Systems Center-Pacific (SSC-PAC); and the NAVAIR Paxtuxent/China Lake. 
These combat system sites replicate, to the maximum extent possible, hardware, 
computer programs, connectivity, and the environment of the ship and aircraft combat 
systems (DEP, 2005). To replicate a Battle Group, the DEP is extended to include the 
combat system sites representing the members of the Battle Group. A combat system 
consists of many tightly integrated key elements. It is the brain of the ship, which 
acquires track data, intelligent data, situational awareness data, and displays these data on 
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its Display console. It interfaces with other subsystems such as radar, target acquisition, 
Cooperative Engagement Capability (CEC), and the Global Command and Control 
System Maritime (GCCS-M), etc (DEP, 2005). A combat system is to be connected, not 
to all sites, but to only the sites that serve as test beds for the required platform combat 
systems. The DEP is thus supposed to be the most cost-effective way to perform 
interoperability testing. 

The remaining of this chapter is organized as follows. Section B discusses the 
Navy systems tested with the DEP; Section C the evolution of DEP testing objectives and 
names; Section D the DEP network architecture; Section E DEP test scheduling; Section 
E DEP staffing; and Section G DEP test results. Section H provides a brief summary of 
the chapter. 

B. NAVY SYSTEMS TESTED WITH THE DEP 

The DEP is intended to test all of the Navy’s systems, specifically combat 
systems as they are the most electronically integrated and their performance is related to 
safety required in firing weapons. DEP testing would involve the Aegis Weapons System 
(AWS), Advanced Combat Direction System (ACDS), radar system, navigation system, 
tactical data link systems, and the Global Command and Control Maritime (GCCS-M) 
system. As a battle group has many ships, which in turn have many combat systems 
(e.g., ACDS, AWS), one single lab (site) would not be able to house all or many systems, 
and labs (sites) across the country would be needed in testing all of these systems as 
illustrated in Table 1 (Coyne, 2006). 

Both Integrated Combat Systems Test Detachment (ICSTD) in San Diego, CA 
and Combat Direction Systems Activity (CDSA) in Dam Neck, VA, house the Advanced 
Combat Direction System (ACDS) system; both the Integrated Weapons Systems Eab 
(IWSE) in Dahlgren, VA, and the Surface Combat Systems Center (SCSC) in Wallops 
Island, VA, house the Aegis Weapons System (AWS); the SPAWAR Systems Center— 
Pacific (SSC-PAC) in San Diego, CA, houses the Global Command and Control System 
Maritime (GCCS-M) systems; both the Patuxent River, MD and SSC-PAC in San Diego, 
CA, house an E-2 Hawkeye (E2C) system; and China Take houses P-18. Systems 
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subjected to DEP testing include a Cooperative Engagement Capability (CEC), Auto 
Identification (ID), Common Data Eink Management System (CDEMS)/Command and 
Control Processor (C2P), Ships Gridlock System/Automatic Correlation (SGS/AC), 
Target Acquisition System Mark-23 (TAS MK-23), Interrogator Set (TPX-42), Integrated 
Automated Detection and Tracking (lADT) System (SYS-2), Eink-4/I I/I6, and Air 
Defense System Integrator (ADSI). Weapons or sensors that are unavailable to a combat 
system are then emulated by computer simulations or a stimulator (SIMS/STIMS). In 
addition, the SIMS/STIMS generates a common environment representing the real world 
and entities therein. The emulated battle group then performs within a controlled, 
repeatable environment under the close scrutiny of engineers and developers (DEP, 
2005). 


Facility/Lab 

Combat Systems 

Numbers of Systems 

Available For Use 

Naval Surface Warfare Center, Port Hueneme Division, 
Detachment San Diego, CA (NSWCPHD - ICSID) 

SSDS MK 2 Mod 1/2 and ACDS ELK 

0/1 

3 (dependent on 
configuration requested) 

Naval Surface Warfare Center, Combat Direction Systems Activity 
Dam Neck, VA (NSWC CDSA) 

SSDS MK 2 Mod 1, ACDS Block 0/1, 
ADSI, and CDS 

3 

Integrated Warfare Systems Laboratory, Dablgren, VA. (IWSL) 

AWS 5/6/7 Baseline 

2 

Center for Surface Combat Systems, Dablgren, VA. (CSCS) 

AWS 2/5/6 Baseline 

2 

Surface Combat Systems Center, Wallops Island, VA. (SCSC) 

AWS 2/5/6/7 Baseline and SSDS 

2 

Naval Air Warfare Center, E2C System Test and Evaluation 
Laboratory, Patuxent River, MD. (NAWCAD PAX - ESTEL) 

Hav^eye 2000 with Mission Computer 
Unit(MCI]) 

1 

Naval Air Warfare Center, Avionics Integration Laboratory, China 
Lake,CA. (NAWCAD-AIL) 

F/A-18 (C, D, and F) Hornet 

2 

Space and Naval Warfare Systems Command, Systems Center San 
Diego, CA. Group LI E2C Test Laboratory. (SSC-SD - E2C) 

Group nE2C (multiple software builds 
supported) 

1 

Space and Naval Warfare Systems Command, Systems Center San 
Diego, CA. Reconilgurable Land-Based Test Site for Global 
Command and Control System - Maritime. (SSC-SD - RLBTS) 

(GCCS-M elements only). GCCS-M 
(multiple hull and system configurations 
possible). 

3 or more 

Naval Surface Warfare Center, Dablgren Division, DEP Operations 
Center, Dablgren, VA. (NSWCDD-DOC) 

Netwnrk/infrastructure, DIS, and 
TACCOMM monitoring^analysis, 
scenario controFbroadcast only 

N/A 

Space and Naval Warfare Systems Command, Systems Center San 
Diego, CA. Systems Integration Facibty. (SSC-SD - SLF) 

ADSI, Data Link (Link-4/11/16), and 
TACCOMM monitoring^analysis. 

N/A 

Space and Naval Warfare Systems Command, Systems Center San 
Diego, CA. Network Operations Center. (SSC-SD-NOC) 

Network/infrastructure operations, 
monitoring, and analysis only 

N/A 

Naval Surface Warfare Center, Corona, CA. (NSWC Corona) 

Voice and TACCOMM monitoring and 
analysis 

N/A 


Table 1. Tabs with Associated Combat Systems (Coyne, 2006) 
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C. EVOLUTION OF DEP TESTING OBJECTIVES AND NAMES 

The original plan of the DEP stopped with “Battle Foree Interoperability Test” 
(BEIT), foeused on the area of Air & Surface Warfare (ASW) and C4I. To conduct 
testing, the DEP used the Navy D-30 process (DEP, 2005) to manage and test software 
before its delivery to the Fleet. To justify future funding to sustain the DEP program, the 
DEP changed its testing objectives and the test name from BEIT to Force Interoperability 
Test (FIT). In addition to focusing on software upgrades, DEP testing now also focused 
on inputs from the Battle Group commanders and sailors. With their inputs, the 
interoperability problems were reproducible in the lab environment and the proposed 
workaround along with the proposed capabilities and limitations were then 
communicated to the Battle Groups. FIT was then changed to Interoperability 
Assessment (lA) in 2005, which was then changed again in 2006 to Basic Platform 
Interoperability Assessments Test (BPIT) and Advanced Platform Interoperability 
Assessments Test (APIT). Subsequently, the name was changed to Interoperability 
Development (I/O DEV) and Interoperability Certification (I/O CERT), which have thus 
far remained unchanged (BFIMS, 2009). 

Whereas the DEP concept remains unchanged, which is to ensure that 
interoperability and software problems are flushed out before delivery to the Fleet, the 
DEP testing requirements changed with the name change. Instead of testing multiple 
Battle Groups, the DEP testing has been concentrated on just one Battle Group with an 
emphasis on certification of new or upgraded software. 

D. DEP NETWORK ARCHITECTURE 

As discussed in Chapter I, the basic idea underlying the DEP is to connect the labs 
(sites) that house the systems identified in Section B of this chapter to emulate a Battle 
Group with its platforms and its operational environment. The inter-site connectivity 
emulates the various communications networks connecting the elements of a strike group 
and “back channel” communications for test coordination and data collection. Computer 
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simulations provide coordinated stimuli to the various platform subsystems at each site. 
The stimuli would be both representative of the operational environment and repeatable 
over successive test runs (DEP, 2005). 

In some DEP-unrelated efforts, connecting the various laboratories using Secret 
Internet Protocol Router Network (SIPRNET) has failed, because of its low-bandwidth, 
its lack of around-the-clock availability, and attending laborious security paperwork. In 
the DEP approach, the laboratories are connected via an Asynchronous Transfer Mode 
(ATM) network using the Defense Information System Network - heading Edge Services 
(DISN-EES). The high-speed ATM network is leased from AT&T. The DEP ATM 
network operates at 10 Mbps, but it could also operate at 45 Mbps (at a relatively low 
cost). Growth to 155 Mbps and beyond is achievable. This network is available 24/7; 
once the network is credited (usually once a year), it can be used at anytime (DEP, 2005). 

E. DEP TESTING SCHEDULING 

The first DEP test was performed on the replicated Eincoln and Truman Battle 
Groups in 1999, whose systems (marked with ‘X’) are listed in Table 2 (Seaver, 2004a). 
This first test was successfully conducted, despite the difficulty associated with the newly 
formed DEP initiative and caused by the unavailability of the labs already scheduled for 
some other events. As the DEP program matured and the fleet requirements changed 
(i.e., testing not only ASW but also C4I systems), the support from some of these labs 
began to vanish; only a handful of the labs are now actually being utilized. Even with the 
smaller number of labs involved, scheduling DEP tests still complicate the schedules and 
planning of the labs involved in DEP testing, as the labs would have to modify their 
schedules, which had been established months in advance. Eurthermore, if the DEP test 
date changed, then the DEP had to re-schedule the labs again, which would create major 
scheduling conflicts for the labs. 

When a few components malfunction during a test, the test has to be repeated. To 
handle the scheduling of tests and retests, the DEP uses a web-based scheduling tool to 
schedule the labs (DEP, 2005). This tool effectively aids the PM and site leads in 
scheduling and re-scheduling the labs. The tool automatically sends an email to the 
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participating sites alerting a need for lab aeeess and an email to the scheduling manager 
indicating the availability of the requested lab. If a schedule conflict arose, the 
scheduling manager would then determine a feasible time for all parties involved in 


testing. 



CVN 

LHA 

CG 

DDG 

DD 

FFG 

E-2 GPn 

F-I4 

COMBAT SYSTEM 

ACDS Blk 0 

ACDS Blk 0 

AWS 

AWS 

CDS 

CDS 

J9VETCBA 

D03B 

C2P 

X 

N/A 

X 

X 

N/A 

N/A 

N/A 

N/A 

SGS/AC 

X 

X 

X 

X 

N/A 

N/A 

N/A 

N/A 

Auto ID 

X 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

TAS MK-23 

X 

X 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

TPX-42(V)8/13/14 

X 

X 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

SYS-2 

X 

X 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

GCCS-M 

X 

X 

X 

X 

X 

X 

N/A 

N/A 

LINK A/J/SAT J 

X 

X 

X 

X 

D 

D 

A/J 

J 

X - Indicates system under test 


Table 2. List of DEP Systems (Seaver, 2004a) 


F. DEP STAFFING 

Staffing has proved to be most challenging as it is related to funding and testing 
tools issues. Sinee DEP testing is not running on a eontinuous basis, the testers are paid 
only when they support the testing. In other words, the testers are paid on a part-time 
basis (i.e., per-test basis). The total funding alloeated to a site manager is based on an 
estimated number of tests to run per year (Avarez, 2006). When the number of tests 
unexpeetedly inerease, the per-test based yearly funding alloeation will not be suffieient 
to eover the testers. 

Many years of DEP testing have shown that a test has never been earned out on 
time. The delay in testing is eaused partly by hardware and software problems and partly 
by the high turnover rate of personnel. Usually, testers would stay in their part-time 
position and leave to aeeept an available full-time position at another organization. 
Often, the new personnel, who replaee the departing testers, have little or no knowledge 
of the eondueted tests and therefore are unable to troubleshoot problems that arise during 
the testing. Although this high turnover rate of personnel has become problematie to the 
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DEP program, the DEP PM has not been willing to pay full-time positions to retain these 
SMEs. The DEP issues of test delay, the pay-per-test eoncept, and a paying alternative 
are diseussed in more detail in Chapter IV. 

G. TEST RESULTS 

Many DEP tests have discovered a few new problems and repeatedly revealed 
identical interoperability problems (Mahon, 2003b). These interoperability problems 
thus have not been fixed. The new problems are reported to the program offices for 
corrections. Some of the interoperability problems have already been documented as 
trouble reports (TRs) in their local databases. If the TRs were high priority which did not 
have the workaround, then corrections would be made during the next software upgrade. 
The TRs have also been presented to the Battle Group personnel. 

Eessons learned and the workaround solutions have been very useful to the Elect 
as the sailors, rather than the engineers and testers, now could use them to figure out the 
causes of interoperability failures and to do the fixing at sea. The lessons learned have 
been also shared with the system developers, but, parenthetically, in the absence of any 
mandating instructions to consider them, the developers largely tend to ignore the lessons 
learned. The DEP program has never implemented the lessons learned in the 
development of systems, because, except for a few engineers called in occasionally for 
root cause analysis, the engineers doing interoperability testing are not the engineers who 
developed the subsystems and also because the system developer would not be around to 
have tests repeated to deal with problems discovered during post-test data analysis and to 
subsequently incorporated the problem resolution in the design of the subsystems. 

H. SUMMARY 

To summarize, the DEP concept is to alleviate of the cost of trying to reproduce 
the interoperability problems encountered during the Elect operations because of the huge 
cost. To be able to reproduce the interoperability problems in the DEP lab environment 
is an incredible accomplishment, but, after several tests, the same problems such as 
duplicated track numbers, dual tracks, etc. have been still observed. In addition, the 
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turnover of the testing personnel has been also a dilemma. Being unable to retain the 
eore SME to support testing and to fix interoperability problems, the DEP might beeome 
one of the programs that have no value added to the war fighter. The DEP should be 
analyzed objeetively for its effectiveness. 
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III. END-TO-END TESTING PROGRAM 


A. INTRODUCTION 

This chapter provides the baekground of and the issues encountered in the E2E 
program. The Navy E2E testing program began in 2007, when the Navy PEO C4I was 
tasked to provide enhaneed, integrated C4I eapabilities through integrated 
eommunieations and information teehnology systems to aid in aehieving battle deeision 
superiority (Miller, 2008). These integrated eommunieations and information teehnology 
systems often make use of new teehnologies. As noted before, when the systems 
eneounter problems at sea, fixing them beeome eostly beeause of the large expenses that 
are ineurred when SMEs are brought from land to ships to eorreet the problems. The idea 
of E2E testing is to leverage the knowledge and existing laboratories on the eampus (not 
aeross the eountry as in the DEP testing program) and within a eommand (not aeross 
different eommands as in the DEP testing program) to ensure that C4I teehnical solutions 
are designed, developed, tested, certified, and delivered to a warship before its 
deployment (Musson, 2008b). In addition, not only does the E2E testing support the 
eertification, but it also supports the developmental testing of multiple programs. 
Requested by PEO C4I to provide support in an effort to eutting sueh expenses, the SSC- 
PAC has begun to build an E2E lab intended to replieate and test shipboard C4I systems 
before their delivery to the Eleet. Building the E2E lab is a ehallenge beeause the 
sponsor pays only a fix amount of funding. The E2E testing PM must therefore analyze 
the E2E tasks earefully, identify the needs the E2E lab must satisfy, and determine the 
minimal cost to build the E2E lab. 

The rest of the ehapter is organized as follows. Seetion B discusses the E2E 
testing concept; Section C E2E testing program eost; Section D E2E test seheduling; 
Seetion E E2E staffing, and Seetion E E2E test results. Seetion G ends with a summary 
of the ehapter. 
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B. END-TO-END TESTING CONCEPT 

SSC-PAC currently manages the E2E testing program and the building of the E2E 
lab. The E2E lab is to house as mueh C4I hardware as possible in order to support the 
E2E testing. Required by the program office and the Eleet, the E2E C4I systems shown 
in Table 3 are to be tested in the E2E testing program and are thus to be housed in the 
E2E lab (Musson, 2008a) . 

Eimited in funding, the E2E testing program plans to employ test engineers from 
the PMWs and to have only a lab manager, a lab teehnieian, network engineers, and 
systems administrators to perform day-to-day lab activities. With all the C4I systems in 
one lab, the idle time observed in the DEP program would be eliminated. Envisioned by 
the PEO C4I, the E2E testing eoneept is intended (Musson, 2008b): 

• To ensure the eomplex system-of-systems capabilities are engineered, tested 
and eertified to work together in a eollaborative environment with 
transpareney and access aeross the PEO C4I portfolio of solutions. 

• To support the development of serviee-oriented arehitecture enterprise, in 
whieh the system is a eollection of eomponents and services developed by 
multiple programs. 

• To provide aeeess to developers, testers, and users of reference 
implementations of systems/eomponents and to make sure that they all 
interact together without having to proeure them. 

• To provide an environment for configuration management, base lining 
interoperability, functionality, and performance. 

• To have the E2E lab operated by an E2E system engineering team and to 
allow events to be staffed by PMs and stakeholders. 

• To have the E2E lab shared by multiple PEO C4I sponsored projeets. 

• To provide mission seenarios and test strings for Unstruetured, Structured, 
Pre-production, and Production Environments operating at multiple 
elassifications (unclassified, eonfidential, and secret). 
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E2E C4I Systems 

Structured Environment 

CFn 

MEDAL 

ADNS 

DMS 

SATCOM 

DCGS-N 

NITES 

ISNS(V) 

DMS Proxy 

ADNS 

DMS 

Radiant Mercury 

USWDSS 

GCCS-J 

ISNS(V) 

DMS Proxy 

TBMCS 

AIS 

GCCS-J w/B 

USWDSS 

GCCS-J 

NECC (Small Footprint) 

C2PC 

GCCS-M 

AIS 

GCCS-Jw/D 

Pre-Production 

CFn 

MEDAL 

C2PC 

GCCS-M 

SATCOM 

DCGS-N 

NITES 

Radiant Mercury 

TBMCS 

NECC (Small Footprint) 




Table 3. E2E C4I Systems (Musson, 2008a) 

The E2E testing PM, interviewed in this research, believes that E2E testing will 
save the PEO C4I money. The reason is that the E2E testing will reduce the probability 
of having at-sea problems with the deployed systems and also because, if the problems 
arose at-sea, troubleshooting might not be as costly as in the case of not having the E2E 
testing capability since the problems would be minor ones. The E2E lab will also support 
the E2E testing systems engineering concept for the PEO C4I portfolio enterprise 
engineering such as development, testing, certification, fielding, and sustainment. The 
purposes of the E2E lab are thus (Musson, 2008b): 

• To provide program transparency, team/cross team collaboration, and to speed 
up delivery. 

• To reduce integration risk. 

• To provide portfolio block and build recommendations and certifications. 

• To implement a distributed, reconfigurable, and dynamic lab environment. 

• To allow for a modular design, plug and play, scheduled, cost-effective 
method of testing. 

Additionally, the E2E testing program is envisioned to connect its C4I systems 
(shown in Table 3) at SPAWAR to other Navy sites’ combat systems, such as NSWC 
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Dahlgren’s Aegis Weapon Systems, NWSC DamNeek’s ACDS Bloek 0/1, PAX River’s 
Airborne systems, and NSWC Rhode Island’s Submarine (Musson, 2008a). 


C. E2E TESTING PROGRAM COST 

A number of programs with laboratory spaees need to move to different loeations 
to make room (spaee) available to the E2E lab. As the E2E testing program has to pay 
for the moves, the total eost of the E2E testing program inereases. The E2E lab was to be 
eompleted in 2008 at the eost of EY08 $8.5M, whieh aeeounts for paying the labs that 
need to move, paying eore engineers, buying the equipment, and building the E2E lab 
(Williams, 2009). More money is now needed in order for the lab to be eompletely built. 
The PEO C4I has promised to provide additional funding to make the E2E testing of the 
Battle Group effort sueeessful, but the amount of funding has not yet been determined. 
The eost of keeping and upgrading the E2E lab to support the E2E testing initiative ean 
potentially be high. 

The E2E testing program is not eurrently in the Program Objeetive Memorandum 
(POM) fund, whieh guarantees funding to the program. Presently, the PEO is taxing the 
PMWs to pay for the E2E testing program (Williams, 2009). 

D. E2E TESTING SCHEDULING 

The E2E lab should be available for all on a first-eome, first-serve basis. A web- 
based seheduling system has been ereated to allow the PM or Test Direetor to sehedule 
his/her tests. Sinee the E2E lab is still under eonstruetion, the availability of the lab has 
not been established and, therefore, exeept for the Eineoln Battle Group test re-seheduled 
in late PY09 by the request of PEO C4I, no tests have been seheduled for EY09 
(Williams, 2009). 

E. E2E TESTING STAFFING 

A lab manager will manage the E2E lab, whieh will be staffed with the eore 
engineers and teehnieians who will install, maintain, and operate the lab (Williams, 
2009). 
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F. 


E2E LAB TEST RESULTS 


Since the E2E lab is still being built, exeept for the eonneetivity testing between 
the E2E lab and the other labs on the SSC-PAC eampus, no full tests have been 
eondueted. The eonneetivity test was successful (Aird, 2009). 

G. SUMMARY 

In summarize, the E2E testing eoneept is a useful eoneept for testing C4I systems 
at the Battle Group integration level before delivering them to the Eleet beeause the Navy 
potentially saves money from not eonducting testing at sea, whieh is costly. The idea 
underlying the E2E testing eoneept is to be proaetive and to eorreet potential problems in 
the labs in order to reduee issues at sea. Aceordingly, savings will be made, and finaneial 
resourees ean then be alloeated to support other efforts. Being able to integrate all the 
C4I systems in the lab environment would be the first aeeomplishment, as it did not oeeur 
in the past. The future will tell whether this eoneept will work as anticipated. 
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IV. ANALYSIS 


A. INTRODUCTION 

As discussed in Chapters II and III, whereas the participant sites receive funding 
on a per-test basis in the DEP testing program, the E2E testing program does not pay 
engineers from the PMWs who conduct tests. Also, as discussed in Chapter III, the E2E 
lab has not been built yet because management did not have a complete plan to establish 
the lab. This ehapter discusses the results of a qualitative analysis of the DEP testing 
information and a quantitative analysis of the DEP testing data in order to provide data to 
aid the DEP testing PM in making decisions on the paying options. It also demonstrates 
the use of goal programming in proving data to support the creation of a plan for building 
the E2E lab. 

The rest of this chapter is organized as follows. Section B discusses the analysis 
of the DEP testing program and Section C the E2E testing program. Seetion D 
summarizes this chapter. 

B. DEP TESTING PROGRAM 

With an emphasis in the area of ASW, the DEP is intended to support only testing 
of Battle Group combat systems. Combat systems have thus far been the subjects of DEP 
testing, beeause NAVSEA, whieh manages not only the DEP program but also the Navy 
ships, heavily emphasizes combat systems and, in partieular, safety assoeiated with firing 
weapons. As the combat systems are tied to C4I systems, the DEP program needs to add 
the latter systems to its testing. Strategically, the DEP wants to expand its testing to C4I 
systems, joint sites and their systems, and commereial sites, sueh as Northup Grumman, 
Eockheed Martin, Boeing, and coalition sites and their systems. However, the current 
DEP budget might not be able to support this expansion (Coyne, 2006). Also, the 
expansion to the eommercial sites has not materialized because the DEP program fails to 
justify the value added by the commercial sites. No new sites have been added. Even if 
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the new sites were needed, funding from the sponsors (NAVSEA and OPNAV) would 
not be available to support the additions. The core sites discussed in Chapter II have thus 
remained in use. 

As pointed out in Chapter II, the DEP participant sites are paid on a per-test basis 
(Mann, 2004). This pay-per-test concept has led to a major problem for the DEP 
program, which is the time loss at the beginning of a test. This time loss, meaning the 
time not used for carrying out the test, is caused by the high personnel turnover rate, 
which in turn is a consequence of the pay-per-test concept. A site that is to carry out a 
test will hire part-time engineers to do the testing. Upon completing the test, these 
engineers, who are no longer funded, move on to other opportunities or to take full-time 
positions elsewhere. The same phenomenon occurs test after test, resulting in a high 
personnel turnover rate. When either a test need be repeated or a new test need be 
performed, newly hired engineers, who have little or no knowledge of the program, spend 
time to learn and to set up the test, resulting in an ineffective use of the allocated testing 
time. Eurthermore, the time loss is also caused by a few sites having problems, as the rest 
of the sites have to wait until the troubled sites are fixed before all the sites can carry out 
the testing. 

An analysis of the data captured from the reviewing of the DEP test reports 
supports the findings described immediately above. The data refer to the uptimes and 
downtimes of the sites, collected between PY02 to PY06 (OTHT, 2008), shown in 
Appendix A. The analysis involves the calculation and an examination of the mean times 
between failures (MTBE) of the sites during this period. The MTBE, which is defined to 
be the total of downtime minus the total of uptime divided by the number of failures 
during the indicated period, is calculated according to (Wikipedia). 


MTBF = ^ 


Downtime — Uptime 
#of Failures 


Based on the data in Appendix A, the calculated MTBE equals to 3 hours and 6 
minutes, which is roughly 38% of an eight-hour test day. The DEP program needs to 
reduce or eliminate this large lost time. The outcomes of the interview with the lead 
(Tran, 2009) indicate that the high turnover rate of the testers at the sites, coupled with 
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new personnel that are not familiar with the DEP testing environment, contribute to the 
substantial amount of time lost per test day. The reasons for this high personnel turnover 
are; 

• Engineers are not fully funded and cannot find a part-time project to make up 
for the short fall. 

• It is difficult to split the task between two or more efforts. 

• Testers unfamiliar with the test cannot get help since the test is conducted 
after hours when the experts have already left for the day. 

• Eormal Risk Mitigation Test (formally called a dry run) has been scaled down 
from one week to one day, and then to none, in order to save money. 

Eigure 2, which displays the delays for the tests repeated during the EY02 to 
EY06 periods, indicates that the first day of each test is the most troublesome day of the 
entire test period, as it takes more than two hours of every 8-hour test day to get the 
systems ready, which would be unacceptable for any test. As a mandatory requirement 
for the sites, the amount of set-up time must be reduced to less than an hour per test day 
to complete the necessary tests. 

The DEP program has been operating since 1999, yet the PM continues to have 
difficulties in determining whether the program should pay the participant sites full-time 
funding or per-test amounts. The quantitative analysis, addressed in Chapter V, is used to 
aid the DEP PM in determining an effective paying option. 

Eigure 2 also shows that in some cases the test day is completely lost, as reflected 
by the peaks of the corresponding curves (e.g., EY02, Test 1, and day 8). With the lost 
time occurring almost daily, the DEP PM should conduct a risk mitigation effort to figure 
out what the sites can do to reduce the lost time. The management should be held 
accountable for not conducting such a risk mitigation effort. If one site were unable to 
support testing, then DEP would be able to utilize the remaining sites. 
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C. END-TO-END TESTING PROGRAM 

Again, contrary to the DEP program, the E2E testing program is concentrated 
mainly on C4I systems. The E2E testing program emphasizes the area of serviees, such 
as Domain Name Server (DNS)/Email/Web; communieation systems, sueh as 
Consolidated Afloat Networks and Enterprise Serviees (CANES)/Advanced Digital 
Network System (ADNS)/Teleport/Network Operation Center (NOC); Information 
Assurance (lA), sueh as Gold DislCVirtual Private Network (VPN)/Cryptos; and network, 
sueh as Satellite Communieations (SATCOMs) & Line of Sight (LOS) & Pierside. In the 
future, E2E testing may be expanded to include systems outside of the C4I arena (E2E, 
2008). 
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As mentioned in Chapter III, the E2E testing has planned to use testers from 
PMWs. The deeision to leverage the PMW testers is flawed beeause (Aird, 2009): 

• The PMW testers would leave after eompletion of a test and would also take 
with them their knowledge, whieh might not be available when needed for 
additional testing. If problems were diseovered from a previous test, the 
PMW testers would not be around to help with the reereation of the problems. 
Even if they eould, the PMW testers would not be able to help immediately 
sinee they might have already been working on different tasks. 

• In regards to the expertise of the testers, if the testers were not seasoned 
testers, then the E2E testing of C4I systems probably would eneounter the 
time loss problems faced by the DEP testing program. 

• Eeveraging testers from different programs would bring a lack of 
accountability as each program would claim effectiveness when, in actuality, 
history has shown otherwise. Also, in the case of failure, a program would 
blame another program. Eurthermore, working across many programs 
requires timely coordination and flexibility of schedule in order to support the 
E2E testing program. In addition, gathering all testers from different 
programs at one time would not be an easy task and an even harder task when 
the schedules slip to the right (Aird, 2009). 

A funding issue arises. The PEO C4I forces the PMWs to support the E2E testing 
program and taxes their programs to pay the E2E testing. The PMWs, however, argue 
that their testing capability would be sufficient (Aird, 2009) to support the Elects and, 
being short on personnel to support additional tasks, want their programs to be left alone. 
The PMWs also believe that the E2E testing program should be POM funded, rather than 
being paid by taxing the PMWs. 

Eurthermore, future funding secured from taxing the PMWs could not always be 
certain. The Assistant E2E testing PM is not sure of how much funding or if any funding 
will be available for EYIO (Williams, 2009). With the funding uncertainty, the E2E 
testing program might not be sustainable in the future. Eor example, the Horizontal 
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Integration program from SPAWAR, which had been funded from taxing PMWs, was 
terminated upon Adm. Gauss’ retirement (Aird, 2009). 

The results of the interview with the E2E testing PM (Williams, 2009) reflect the 
uncertainties the PM has experienced in regards to building the E2E lab with the 
available funding. The first E2E testing of the Eincoln Battle Group, which was to occur 
in December 2008 (E2E, 2008), did not take place, because the E2E lab was not ready. 
In fact, the E2E lab is currently not ready to support any testing, and, because of funding 
uncertainty, the PM does not know when it will be up and running. In addition, the PM 
does not have a complete plan to establish the E2E lab, which has contributed to the 
delay in getting the E2E lab ready. The goal programming approach, addressed in 
Chapter V, is used to aid the E2E testing PM in determining an effective implementation 
plan to build the lab. 

D. SUMMARY 

Prom the review of the existing literature on the DEP and E2E testing programs 
and from the interviews with the respective PMs and leads, (1) the DEP pay-per-test 
concept has led to a major problem of testing inefficiency, namely, the loss of test time to 
setting up the labs by inexperienced part-time engineers, and (2) the E2E testing program 
does not have a complete plan to build the E2E lab. 
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V. COMPARATIVE ANALYSIS AND GOAL PROGRAMMING 


A. INTRODUCTION 

As noted in Chapter II, the DEP program has been operating since 1999, yet the 
PM continues to have difficulties in determining whether the program should pay the 
participant sites full-time funding or per-test amounts. Rather than paying for full-time 
testers, the DEP PM has chosen the pay-per-test method. The comparative analysis 
discussed in this chapter aims at providing data to aid the DEP PM in selecting the best 
paying option for the DEP testing program. 

As observed in Chapter III, the E2E lab has not been built. One of the causes of 
this debacle is the E2E testing PM has failed to determine the funding needed to build the 
E2E lab and to get it ready to support testing. The goals for E2E testing have not been 
established and spelled out clearly in the plan to build the E2E lab. To remedy this 
undesirable situation, this thesis uses goal programming (GP) (Ragsdale, 2007) to 
establish the goals of the E2E lab in the presence of funding constraints. 

The rest of this chapter is organized as follows. Section B presents the results of 
the comparative analysis of the two paying approach: pay-per-test and pay-full-time. 
Section C discusses and demonstrates the goal programming approach to aiding in the 
creation of a plant for building the E2E lab. Section D provides a brief summary of the 
chapter. 

B. COMPARATIVE ANALYSIS 

Table 4 shows seven distributed test sites involved in interoperability certification 
testing: Dahlgren, Dam Neck, NSWC San Diego, NAVAIR, SSC-PAC TADIE, SSC- 
PAC GCCS-M, and Wallops Island. As discussed in Chapter II, NAVSEA has two 
paying options for the DEP testing: paying the sites a fixed amount of money to retain 
full-time testers or paying the sites on a per-test basis. The first option is not necessarily 
cost-effective, as the sites will constantly deplete resources, specifically funding, whether 
they are doing testing, or staying idle. The latter paying option might save NAVSEA 
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money, but as discussed in Chapter IV, it results in 38% of the testing lost to the initial 
test set-up. Constrained by the available budget, the DEP program must decide between 
the two paying options. To aid the DEP program in making an informed decision, a 
comparative analysis is performed this research, in which the DEP data are collected and 
analyzed, to answer the question as to which pay method—pay-per-test or pay-full- 
time—the DEP program should use. 

The comparison is made for all the years of DEP testing from EYOl to PY09. 
Eigure 3 shows the number of tests carried out in those years, which ranges from three to 
eight per year. It is noted that less than six tests were carried out in EYOl and EY05, and 
at least six tests were carried in the remaining years. Eor the purpose of illustration, 
EY09, during which eight tests are conducted, is considered for comparing the costs 
resulting from using the two different methods of payment. The EY09 data shown in 
Table 4 are input to the comparative analysis. The data (BFIMS, 2009) are: 

• The maximum number of engineers at a test site, which is two, to support a 
test from pre-test through the post-test, and their hourly salary. 

• The total test time of two weeks, implying 160 hours for two engineers. 

• The time allocated for attending test planning working group meetings 
(TPWG), working on test plan/procedures, and reviewing test 
plan/procedures, which totals two and half weeks, implying 200 hours for 
two engineers. 

• The time for pre-test checkout prior to the actual test, which is 32 hours for 
two engineers. 

• The time allocated for post-test draft/final test report and trouble reports input 
to the database, which is two weeks, implying 160 hours for two engineers. 

• The total lost time per test incurred by two engineers, which varies from test 
to test. 

• The total man-hours per year, which is 1750. 
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Table 4. NAVSEA DEP Program PY09 Data 
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# of Tests Conducted From FY01-FY09 


■ # of Tests 



2001 2002 2003 2004 2005 2006 2007 2008 2009 


Fiscal Year 

Figure 3. Number of Tests Conducted From FY01-FY09 

A calculation using the data in Table 4 shows that, for conducting eight tests 
during FY09 with two engineers, using the pay-per-test method costs approximately 
$3.8M whereas using the full-time payment method costs about $2.7M 

The same calculation is carried out to obtain the total costs from FYOl to FY09 
(BFIMS, 2009), using all but the hourly costs in Table 5 and accounting for the different 
number of tests from year to year. Figure 4, displaying the costs as a function of the 
number of tests incurred from using the two paying methods, indicates that if the number 
of tests per year is not greater than six, the pay-per-test method is cheaper than the pay- 
full-time option. If more than six tests per year are carried out, the full-time is cheaper 
than the pay-per-test option. Note that the comparison is made with respect to cost only. 
Factors other than cost that may play in the decision making process are not considered in 
this thesis. 

Furthermore, as six or less tests are conducted for only two out of the nine years, 
the pay-full-time option would have collectively saved money during FY01-FY09. In 
this case, the core experienced engineers would have been likely retained and the lost 
time would have been reduced, if not avoided. Based on Figure 4, the DEP program 
would have saved roughly $1.2 M over the FY01-FY09 if the pay-full-time option had 
been instituted. Thus, in the long run, if more than six tests are conducted, savings will 
be made with the pay-full-time option. 
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Table 5. FY01-FY09 Pay-per-Test vs. Pay-Full-Time 
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Fiscal Year 01-09 Pay-Per-Test and Pay-Full-Time Comparison 


□ Pay Per Test 0 Pay Full-Time 



Figure 4. FY01-FY09 Pay-per-Test vs. Pay-Full-Time 

C. E2E TESTING PROGRAM 

The E2E lab, whieh was to conduct testing of a battle group by November 2008, 
is still in the planning phase. The E2E testing program is still determining the funding 
needed to build the E2E lab and to get it ready to support testing. The funding issue is 
elaborated in Chapter III. The level of funding depends on the needs (or goals) of the 
E2E lab, which, as part of the E2E lab building plan, must be determined and supported 
by a rigorous analysis. 

As goals, the lab needs to have desk/chairs, racks, conference rooms, and space. 
Specifically, for the purpose of illustration, the E2E lab building scenario considered in 
this thesis requires approximately 40 desks and chairs, 20 computer racks, and 2 
conference rooms. The desks/chairs, racks, conference rooms, and the lab require 100, 
100, 1,050, and 8,000 /f^of space, respectively. Eurthermore, the desks/chairs, racks, and 
conference rooms each cost $5,000, $7,000, and $156,000, respectively. The budget 
available for the E2E lab is $650,000. 

Regarding meeting these goals, underachieving the first three goals related to the 
number of desks/chairs, racks, conference rooms would be undesirable, but 

overachieving these goals would be acceptable. Also, underachieving or overachieving 
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the goal of adding 8,000 ft^ would be undesirable. Finally, only spending more than 
$650,000 would be undesirable. (The objeetive funetion defined below refleets this 
eonsideration.) 

The question to answer is: Can the E2E testing program deliver the E2E lab that 
will meet these goals for sueh a budget? The approaeh to answering this question is to 
formulate the E2E testing optimal planning problem as a goal programming problem and 
to solve the resulting goal programming problem using Exeel Solver. 

The E2E testing goal programming problem is now formulated. Eet stand for 

the number of desks and ehairs, Xjthe number of eomputer raeks, and and the 
number of eonferenee rooms. Eet vj, Vj, V 3 , V 4 , and vj be, respeetively, the target 
values of the number of desks and ehairs, the number of eomputer raeks, the number of 
eonferenee rooms, the total square footage, and the total eost, whieh are 40, 20, 2, 
8,000 ft^, and $650,000, respeetively. These values are not derived from a rigorous 
analysis, and they might not be possibly met. To aeeount for this unfounded 
restrietiveness, let the so-ealled deviational variables d“and represent the 

amounts by whieh the goals ean deviate from their respeetive target values. Speeifieally, 
represents the amount by whieh the f* goal’s target value is underaehieved, and 

represents the amount by whieh the i‘^ goal’s target value is overaehieved. 

The eonstraints are thus: 

+«i2^2 +«;3^3 =Vi 

X. e Z and X. > 0 

d;,dt>0 

in whieh, for the E2E testing problem at hand, the eoeffieients = a 12 = = 1, 

«,4 = = 0, for / = 1,2, and 3, and « 4 i =«42 =100/?^, 0:43 = 1,050/t^, and 

= $ 5 , 000 , 0:52 = $^’000’^iid «53 = $156,000. Z denotes the set of integers; 
X^ G Z and X. > 0 together mean that X^ are nonnegative integers. 

The objeetive funetion is defined as 
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z = 



+ 



in which the weight eoeffieients, w andw^,; =l, -,5, represent numerie eonstants that ean 

be assigned values to weight the various deviational variables. The objeetive funetion is 
thus a weighted pereentage deviation. A variable that represents a highly undesirable 
deviation from a partieular goal is assigned a relatively large weight. A variable that 
represents a neutral or desirable deviation from a partieular goal is assigned a weight of 0 
or lower than 0. Analysis of the goal programming begins with 
= W 4 = W 4 = = 1 and all other weights being 0 . 

The E2E testing goal programming problem is thus; Minimize z subjeet to the 
eonstraints above. The solution to this E2E testing goal programming problem is 
obtained, using Exeel Solver (Ragsdale, 2007), and is eaptured in Table 6 . 

As indieated by Table 6 , the planned eost (budget), whieh is $650,000, exeeeds 
the optimal eost, whieh is $647,000, by $3,000. Whereas money savings are realized, the 
targeted number of desks and ehairs is short by one. As aforementioned, this negligible 
shortage is aeeeptable. 


E2E Goal Programming 













Problem Data 

Desk & Chair 

Rack 

Conference Room 



Square Footage 

100 

100 

1050 



Cost 

5000 

7000 

156000 









Goal Contraints 

Desk & Chair 

Rack 

Conference Room 

Sq. Ft. 

Cost 

Actual Amount 

39 

20 

2 

8000 

$647,000 

Under + 

1 

0 

0 

0 

$3,000 

Over- 

0 

0 

0 

0 

0 

Goal = 

40 

20 

2 

8000 

$650,000 

Target Value 

40 

20 

2 

8000 

$650,000 


Table 6 . E2E Testing Goal Programming Solver Results 
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The E2E testing PM ean thus use this formulated problem with different budgets 
and numbers of items, sueh as desk/ehairs, raeks, eonferenee rooms, and spaee, ete., to 
eome up with the best available minimum eost to support the E2E testing effort. 
Additionally, the E2E testing PM ean use this method to justify the funding for the 
eurrent year and future funding. 

D. SUMMARY 

In summary, the eomparative analysis indieates that paying the DEP sites full¬ 
time will save the DEP program if it eonduets six or more tests per year. If the number of 
tests is less than six per year, then DEP program has the flexibility in seleeting either of 
the paying options. An additional advantage of paying the DEP sites on a full-time basis 
is that the DEP sites will likely be able to retain experieneed testers for a long period of 
time and henee to be able to keep the eontinuity and eohesiveness of the team. 

Goal programming ean be used effeetively to aid in establishing a plan for 
building an E2E lab for the Navy E2E testing. It provides a rigorous approaeh to 
determining the goals of the E2E lab in a timely fashion (henee, saving money) that ean 
meet budgetary eonstraints. It ean also be used in support of planning for programs other 
than testing as well as for many multi-goal problems eneountered in systems engineering. 
Results obtained with this goal programming approaeh are used in deeision making, in 
assessing the feasibility of aehieving the goals, and also in knowing how mueh funding is 
required to meet the goals. 
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VI. CONCLUSION 


Dealing with test and evaluation of combat and C4I systems in the U.S. Navy, this 
thesis focuses specifically on two separate Navy testing programs managed by two 
different commands; the DEP testing program, run by NAVSEA, and the E2E testing 
program, currently being formed by SPAWAR. 

During deployments and Battle Group exercises. Battle Group systems 
encountered interoperability problems such as erroneous dual/multiple track designations, 
misidentification/track identity conflicts, report responsibility conflicts, friendly tracks 
displayed as unknown/pending, tracks dropped without operator action, different track 
identities of different ships, etc. When these interoperability problems occurred at sea, 
fixing them was costly because of the high cost incurred in bringing SMEs from land to 
ships to correct the problems. In an effort to prevent these costly problems, the DEP 
testing program was formed in June 1998 to support the evaluation of the interoperability 
of Battle Group systems and also to aid in design decision making early in the systems 
acquisition process. The DEP consists of shore-based combat system sites, which are 
currently paid on a per-test basis, rather than on a retaining full-time basis. 

The purpose of the E2E testing is to evaluate integrated capabilities of shipboard 
C4I systems for interoperability. The shipboard C4I systems need be tested before their 
delivery to the warships. Not only does E2E testing support the certification, but it also 
supports the developmental testing of multiple programs. In the future, the E2E testing 
may be expanded to include systems outside of the C4I arena. In the E2E testing 
concept, formulated in 2007, a single E2E systems engineering laboratory, which is yet 
to be built, replicates and tests these C4I systems. Eimited in funding, the E2E testing 
program plans to employ test engineers from the PMW organizations. A challenge faced 
by the E2E testing program is building and getting the E2E lab ready for testing. Two 
factors contributing to this challenge are uncertain availability of funding for building the 
E2E lab and the lack of a comprehensive plan to establish the E2E lab. 

With a focus on the issue of cost-effective implementation of the DEP and E2E 
testing programs, the research performed in this thesis involves (1) conducting a review 
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of the past and current test results of the distributed, E2E programs and their archived 
documents, related distributed and E2E testing studies, and other pertinent related 
material such as test report, test plans, and test procedures; (2) developing interview 
questionnaires directed at managers of the DEP and E2E testing programs, (3) conducting 
interviews with the DEP and E2E testing PMs, test directors, and functional leads and 
process literature and interview results; and (4) performing qualitative and quantitative 
analysis and goal programming to aid in determining the optimal costs of carrying out 
these testing methods. The findings from this research follow. 

The DEP program currently pays the DEP participant sites on a per-test basis. As 
a result, the allocated testing time is not fully used for testing; roughly 38% of the 
allocated testing time is lost at the beginning of a test. Another paying option for the 
DEP program to consider is paying the DEP participant sites a fixed amount of money to 
retain full-time engineers. The testing cost data from EYOl to PY09 indicate that 
conducting eight tests during PY09 with two engineers, using the pay-per-test method 
costs approximately $3.8M, whereas using the full-time payment method costs about 
$2.7M, and that if the number of tests per year is not greater than six, the pay-per-test 
method is cheaper than the pay-full-time option. As six or less tests were conducted for 
only two out of the nine years (EYOl and EY05), the pay-full-time option would have 
collectively saved money during EY01-EY09. Thus, in the long run, if more than six 
tests are conducted, savings will be made with the pay-full-time option. 

Using testers from the PMWs to conduct the E2E testing would possibly lead to 
the time loss problem faced by the DEP program, as the PMW testers would leave after 
completion of the testing and would also take with them their knowledge; a lack of 
accountability as different programs from which testers come would claim effectiveness 
when, in actuality, history has shown otherwise; finger pointing in the case of failure; 
difficulty in timely coordination and flexibility of schedule in order to support the E2E 
testing program; and possibly not meeting the schedule. Rather being POM funded, the 
E2E testing program secures its funding from taxing the PMWs. As such funding is 
uncertain, the E2E testing program might not be sustainable in the future. Eunding 
uncertainty and the lack of a complete plan to establish the E2E lab contribute to the 
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delay in building and getting the E2E lab ready for testing. Einally, goal programming 
ean be used as a rigorous approaeh to determining the goals of the E2E lab in a timely 
fashion (henee, saving money) that ean meet budgetary eonstraints. Eor the seenario 
treated in this researeh, the planned eost for building the E2E lab, whieh is $650,000, 
exeeeds the optimal eost obtained with the goal programming approaeh, whieh is 
$647,000, by $3,000. Whereas money savings are realized, the targeted number of desks 
and ehairs is short by one. This negligible shortage is aeeeptable. The E2E testing 
program ean use this approaeh to justify the funding for the eurrent year and future 
funding. 

It is reeommended that (1) the DEP sites be funded on a pay-per-test basis if no 
more than six tests are performed in a year and on a full-time basis otherwise; (2) the eore 
engineers be retained to mn DEP tests; (3) optimization teehniques in general and goal 
programming in partieular be employed in planning for and, speoifieally, in formulating a 
plan for building the E2E lab; and (4) rigorous approaehes—analytieal and/or 
optimization teehniques—be employed from the beginning of testing programs to plan 
for and implement the testing programs. 

Some areas for further researeh inelude (1) extending the analysis and 
optimization teehniques diseussed in this thesis to determine the optimal methods of 
eondueting tests with eonstraints on resourees sueh as laboratories, personnel, sehedules, 
ete. and (2) using the approaeh espoused in this thesis as a foundation for additional 
researeh and studies of the joint distributed testing/eoalition distributed testing. 
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APPENDIX A. DEP TEST DELAY DATA (EY02-EY06) 


FY02 





Test 1 


Start 

Loading 

(h:mm) 

End 

Loading 
(Start to 
test) 
(h:mm) 

Different 

(h:mm) 

Week 1 

Day 1 

21:00 

2:22 

5:22 


Day 2 

21:00 

0:17 

S:17 


Day S 

21:00 

22:12 

1:12 


Day 4 

21:00 

1:25 

4:25 


Day 5 

21:00 

22:28 

1:28 

Week 2 

Day 1 

21:00 

1:20 

4:20 


Day 2 

21:00 

2S:15 

2:15 


Day S 

21:00 

4:20 

7:20 


Day 4 

21:00 

21:15 

0:15 


Day 5 

21:00 

22:10 

1:10 

Average 




3:06 

Test 2 





Week 1 

Day 1 

21:00 

0:S0 

3:30 


Day 2 

21:00 

2S:20 

2:20 


Day S 

21:00 

2S:02 

2:02 

Week 2 

Day 1 

21:00 

0:01 

3:01 


Day 2 

21:00 

0:02 

3:02 


Day S 

21:00 

22:11 

1:11 

Weeks 

Day 1 

21:00 

2S:09 

2:09 


Day 2 

21:00 

22:SS 

1:33 


Day S 

21:00 

22:40 

1:40 


Day 4 

21:00 

21:51 

0:51 

Average 




2:07 

Test 3 





Week 1 

Day 1 

21:00 

5:00 

8:00 


Day 2 

21:00 

2:01 

5:01 


Day S 

21:00 

1:0S 

4:03 

Week 2 

Day 1 

21:00 

5:00 

8:00 


Day 2 

21:00 

22:05 

1:05 


Day S 

21:00 

2S:12 

2:12 

Weeks 

Day 1 

21:00 

4:15 

7:15 


Day 2 

21:00 

22:02 

1:02 


Day S 

21:00 

2S:11 

2:11 


Day 4 

21:00 

22:12 

1:12 

Average 




4:00 

Test 4 
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Week 1 

Day 1 

21:00 

1:07 

4:07 


Days 

21:00 

0:57 

S:57 


Day S 

21:00 

22:15 

1:15 

Week 2 

Day 1 

21:00 

22:S1 

1:S1 


Days 

21:00 

22:16 

1:16 


Day S 

21:00 

5:00 

8:00 

Weeks 

Day 1 

21:00 

2S:0S 

2:0S 


Days 

21:00 

0:0S 

S:0S 


Day S 

21:00 

5:00 

8:00 


Day 4 

21:00 

1:0S 

4:0S 

Average 




3:43 

Test 5 





Week 1 

Day 1 

21:00 

22:22 

1:22 


Days 

21:00 

0:01 

3:01 


Day S 

21:00 

22:01 

1:01 


Day 4 

21:00 

2S:05 

2:05 

Weeks 

Day 1 

21:00 

0:02 

3:02 


Days 

21:00 

2S:01 

2:01 


Day S 

21:00 

0:07 

3:07 

Weeks 

Day 1 

21:00 

0:10 

3:10 


Days 

21:00 

0:05 

3:05 


Day S 

21:00 

0:00 

3:00 

Average 




2:29 

Test 6 





Week 1 

Day 1 

21:00 

2S:55 

2:55 


Days 

21:00 

2S:15 

2:15 


Day S 

21:00 

2S:42 

2:42 


Day 4 

21:00 

0:55 

3:55 


Day 5 

21:00 

2S:50 

2:50 

Weeks 

Day 1 

21:00 

0:52 

3:52 


Days 

21:00 

0:57 

3:57 


Day S 

21:00 

0:50 

3:50 


Day 4 

21:00 

2S:55 

2:55 


Day 5 

21:00 

2S:S0 

2:30 

Average 




3:10 


FY03 





Test 1 


Start 

Loading 

End 

Loading 
(Start to 
test) 

Different 

Week 1 

Day 1 

21:00 

0:05 

3:05 


Day 2 

21:00 

22:35 

1:35 
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Day 3 

21:00 

22:04 

1:04 


Day 4 

21:00 

22:49 

1:49 

Week 2 

Day 1 

21:00 

0:44 

3:44 


Days 

21:00 

0:34 

3:34 


Day 3 

21:00 

23:57 

2:57 

Weeks 

Day 1 

21:00 

22:33 

1:33 


Days 

21:00 

22:31 

1:31 


Day 3 

21:00 

23:15 

2:15 

Average 




2:18 

Test 2 





Week 1 

Day 1 

21:00 

23:03 

2:03 


Days 

21:00 

22:12 

1:12 


Day 3 

21:00 

22:27 

1:27 


Day 4 

21:00 

0:01 

3:01 


Day 5 

21:00 

0:02 

3:02 


Day 1 

21:00 

1:08 

4:08 


Days 

21:00 

23:00 

2:00 


Day 3 

21:00 

5:00 

8:00 


Day 4 

21:00 

0:10 

3:10 


Day 5 

21:00 

0:10 

3:10 

Average 




2:24 

Test 3 





Week 1 

Day 1 

21:00 

22:32 

1:32 


Days 

21:00 

0:10 

3:10 


Day 3 

21:00 

22:48 

1:48 


Day 4 

21:00 

22:31 

1:31 


Day 5 

21:00 

0:07 

3:07 

Weeks 

Day 1 

21:00 

22:45 

1:45 


Days 

21:00 

0:23 

3:23 


Day 3 

21:00 

21:38 

0:38 


Day 4 

21:00 

0:05 

3:05 


Day 5 

21:00 

21:59 

0:59 

Average 




2:06 

Test 4 





Week 1 

Day 1 

21:00 

1:07 

4:07 


Days 

21:00 

0:07 

3:07 


Day 3 

21:00 

22:15 

1:15 


Day 4 

21:00 

22:51 

1:51 


Day 5 

21:00 

23:16 

2:16 

Weeks 

Day 1 

21:00 

1:30 

4:30 


Days 

21:00 

23:03 

2:03 


Day 3 

21:00 

0:23 

3:23 


Day 4 

21:00 

0:10 

3:10 


Day 5 

21:00 

1:23 

4:23 

Average 




3:00 

Test 5 





Week 1 

Day 1 

21:00 

23:02 

2:02 


45 




Days 

21:00 

0:11 

3:11 


Day 3 

21:00 

22:01 

1:01 


Day 4 

21:00 

22:45 

1:45 


Day 1 

21:00 

1:02 

4:02 


Days 

21:00 

22:31 

1:31 


Day 3 

21:00 

0:47 

3:47 


Day 1 

21:00 

0:20 

3:20 


Day 2 

21:00 

0:48 

3:48 


Day 3 

21:00 

0:38 

3:38 

Average 




2:48 

Test 6 





Week 1 

Day 1 

21:00 

5:00 

8:00 


Day 2 

21:00 

2:11 

5:11 


Day 3 

21:00 

1:33 

4:33 

Week 2 

Day 1 

21:00 

4:10 

7:10 


Day 2 

21:00 

22:35 

1:35 


Day 3 

21:00 

23:12 

2:12 

Weeks 

Day 1 

21:00 

4:00 

7:00 


Days 

21:00 

22:42 

1:42 


Day 3 

21:00 

23:32 

2:32 


Day 4 

21:00 

22:12 

1:12 

Average 




4:06 

Test 7 





Week 1 

Day 1 

21:00 

4:00 

7:00 


Day 2 

21:00 

2:11 

5:11 


Day 3 

21:00 

1:33 

4:33 


Day 4 

21:00 

4:00 

7:00 


Day 5 

21:00 

22:35 

1:35 

Weeks 

Day 1 

21:00 

23:12 

2:12 


Day 2 

21:00 

2:10 

5:10 


Day 3 

21:00 

22:42 

1:42 


Day 4 

21:00 

23:32 

2:32 


Day 5 

21:00 

22:12 

1:12 

Average 




3:48 


FY04 





Test 1 


Start 

Loading 

End 

Loading 
(Start to 
test) 

Different 

Week 1 

Day 1 

21:00 

2:46 

5:46 


Days 

21:00 

23:10 

2:10 


Day 3 

21:00 

5:00 

8:00 
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Day 4 

21:00 

2S:55 

2:55 


Day 5 

21:00 

2S:10 

2:10 

Week 2 

Day 1 

21:00 

2:44 

5:44 


Day 2 

21:00 

1:19 

4:19 


Day S 

21:00 

0:15 

S:15 


Day 4 

21:00 

0:20 

S:20 


Day 5 

21:00 

0:24 

S:24 

Average 




4:06 

Test 2 





Week 1 

Day 1 

21:00 

22:02 

1:02 


Day 2 

21:00 

22:42 

1:42 


Day S 

21:00 

2S:12 

2:12 


Day 4 

21:00 

0:01 

S:01 


Day 5 

21:00 

0:02 

S:02 

Week 2 

Day 1 

21:00 

22:11 

1:11 


Day 2 

21:00 

2S:09 

2:09 


Day S 

21:00 

22:SS 

1:SS 


Day 4 

21:00 

0:10 

S:10 


Day 5 

21:00 

0:0S 

S:0S 

Average 




2:12 

Test 3 





Week 1 

Day 1 

21:00 

1:07 

4:07 


Day 2 

21:00 

1:01 

4:01 


Day S 

21:00 

0:27 

S:27 


Day 4 

21:00 

22:15 

1:15 

Week 2 

Day 1 

21:00 

22:51 

1:51 


Day 2 

21:00 

2S:16 

2:16 


Day S 

21:00 

5:00 

8:00 

Weeks 

Day 1 

21:00 

22:0S 

1:0S 


Day 2 

21:00 

5:00 

8:00 


Day S 

21:00 

5:00 

8:00 

Average 




4:12 

Test 4 





Week 1 

Day 1 

21:00 

1:07 

4:07 


Day 2 

21:00 

0:47 

S:47 


Day S 

21:00 

22:15 

1:15 

Week 2 

Day 1 

21:00 

22:51 

1:51 


Day 2 

21:00 

2S:16 

2:16 


Day S 

21:00 

5:00 

8:00 

Weeks 

Day 1 

21:00 

2S:0S 

2:0S 


Day 2 

21:00 

0:2S 

S:2S 


Day S 

21:00 

5:00 

8:00 


Day 4 

21:00 

0:2S 

S:2S 

Average 




3:48 

Test 5 





Week 1 

Day 1 

21:00 

0:15 

3:15 


Day 2 

21:00 

2S:20 

2:20 


47 




Day 3 

21:00 

23:20 

2:20 

Week 2 

Day 1 

21:00 

0:51 

3:51 


Day 2 

21:00 

0:55 

3:55 


Day 3 

21:00 

22:51 

1:51 

Weeks 

Day 1 

21:00 

23:19 

2:19 


Day 2 

21:00 

22:33 

1:33 


Day 3 

21:00 

23:50 

2:50 


Day 4 

21:00 

21:51 

0:51 

Average 




2:30 

Test 6 





Week 1 

Day 1 

21:00 

1:07 

4:07 


Day 2 

21:00 

0:57 

3:57 


Day 3 

21:00 

22:15 

1:15 


Day 4 

21:00 

22:01 

1:01 


Day 5 

21:00 

23:10 

2:10 

Weeks 

Day 1 

21:00 

0:15 

3:15 


Day 2 

21:00 

23:03 

2:03 


Day 3 

21:00 

0:13 

3:13 


Day 4 

21:00 

1:00 

4:00 


Day 5 

21:00 

1:05 

4:05 

Average 




2:54 


FY05 





Test 1 


Start 

Loading 

End 

Loading 
(Start to 
test) 

Different 

Weeki 

Day 1 

21:00 

3:55 

6:55 


Days 

21:00 

23:25 

2:25 


Day 3 

21:00 

23:04 

2:04 


Day 4 

21:00 

23:55 

2:55 

Weeks 

Day 1 

21:00 

2:54 

5:54 


Day 2 

21:00 

1:49 

4:49 


Day 3 

21:00 

23:57 

2:57 

Weeks 

Day 1 

21:00 

23:43 

2:43 


Day 2 

21:00 

23:35 

2:35 


Day 3 

21:00 

23:45 

2:45 

Average 




3:36 

Test 2 





Weeki 

Day 1 

21:00 

23:28 

2:28 


Day 2 

21:00 

22:42 

1:42 


Day 3 

21:00 

22:31 

1:31 


Day 4 

21:00 

0:11 

3:11 


Day 5 

21:00 

0:02 

3:02 

Weeks 

Day 1 

21:00 

0:08 

3:08 


48 




Day 2 


23:30 

2:30 


Day 3 

21:00 

5:00 

8:00 


Day 4 

21:00 

0:15 

3:15 


Day 5 

21:00 

0:10 

3:10 

Average 




2:30 

Test 3 





Week1 

Day 1 

21:00 

1:30 

4:30 


Day 2 

21:00 

0:40 

3:40 


Day 3 

21:00 

5:00 

8:00 


Day 4 

21:00 

23:31 

2:31 


Day 5 

21:00 

1:47 

4:47 

Week 2 

Day 1 

21:00 

23:37 

2:37 


Day 2 

21:00 

0:23 

3:23 


Day 3 

21:00 

23:38 

2:38 


Day 4 

21:00 

0:45 

3:45 


Day 5 

21:00 

23:39 

2:39 

Average 




4:00 


FY06 





Test 1 


Start 

Loading 

End 

Loading 
(Start to 
test) 

Different 

Week 1 

Day 1 

21:00 

0:04 

3:04 


Day 2 

21:00 

22:00 

1:00 


Day 3 

21:00 

22:09 

1:09 


Day 4 

21:00 

0:13 

3:13 


Day 5 

21:00 

0:03 

3:03 

Week 2 

Day 1 

21:00 

22:32 

1:32 


Day 2 

21:00 

0:15 

3:15 


Day 3 

21:00 

21:59 

0:59 


Day 4 

21:00 

0:03 

3:03 


Day 5 

21:00 

0:13 

3:13 

Average 




2:12 

Test 2 





Week 1 

Day 1 

21:00 

0:15 

3:15 


Day 2 

21:00 

22:25 

1:25 


Day 3 

21:00 

22:04 

1:04 


Day 4 

21:00 

22:49 

1:49 

Week 2 

Day 1 

21:00 

0:34 

3:34 


Day 2 

21:00 

0:04 

3:04 


Day 3 

21:00 

23:17 

2:17 

Week 3 

Day 1 

21:00 

22:53 

1:53 


Day 2 

21:00 

22:31 

1:31 


Day 3 

21:00 

23:15 

2:15 


49 





Average 




2:12 

Test 3 





Week 1 

Day 1 

21:00 

22:S2 

1:S2 


Day 2 

21:00 

1:S0 

4:S0 


Day S 

21:00 

21:48 

0:48 


Day 4 

21:00 

2S:S1 

2:S1 


Day 5 

21:00 

1:47 

4:47 

Week 2 

Day 1 

21:00 

22:57 

1:57 


Day 2 

21:00 

0:2S 

S:2S 


Day S 

21:00 

21:S8 

0:S8 


Day 4 

21:00 

0:45 

S:45 


Day 5 

21:00 

21:59 

0:59 

Average 




2:30 

Test 4 





Week 1 

Day 1 

21:00 

0:07 

3:07 


Day 2 

21:00 

0:57 

3:57 


Day S 

21:00 

22:15 

1:15 


Day 4 

21:00 

22:51 

1:51 


Day 5 

21:00 

2S:16 

2:16 

Week 2 

Day 1 

21:00 

1:00 

4:00 


Day 2 

21:00 

2S:0S 

2:03 


Day S 

21:00 

0:2S 

3:23 


Day 4 

21:00 

1:00 

4:00 


Day 5 

21:00 

1:15 

4:15 

Average 




3:00 

Test 5 





Week 1 

Day 1 

21:00 

2S:02 

2:02 


Day 2 

21:00 

0:00 

3:00 


Day S 

21:00 

22:01 

1:01 


Day 4 

21:00 

22:45 

1:45 

Week 2 

Day 1 

21:00 

0:05 

3:05 


Day 2 

21:00 

22:S1 

1:31 


Day S 

21:00 

0:07 

3:07 

Weeks 

Day 1 

21:00 

0:10 

3:10 


Day 2 

21:00 

0:05 

3:05 


Day S 

21:00 

0:20 

3:20 

Average 




2:30 

Test 6 





Week 1 

Day 1 

21:00 

2S:02 

2:02 


Day 2 

21:00 

22:45 

1:45 


Day S 

21:00 

2S:05 

2:05 

Week 2 

Day 1 

21:00 

22:55 

1:55 


Day 2 

21:00 

22:S5 

1:35 


Day S 

21:00 

2S:12 

2:12 

Weeks 

Day 1 

21:00 

S:00 

6:00 


Day 2 

21:00 

22:45 

1:45 


Day S 

21:00 

2S:S2 

2:32 


50 




Day 4 

21:00 

23:12 

2:12 

Average 




2:24 

Test 7 





Week 1 

Day 1 

21:00 

0:15 

3:15 




23:20 

2:20 




23:57 

2:57 

Week 2 

Day 1 

21:00 

0:01 

3:01 


Day 2 

21:00 

0:10 

3:10 


Day 3 

21:00 

22:51 

1:51 

Weeks 

Day 1 

21:00 

23:09 

2:09 


Day 2 

21:00 

22:33 

1:33 


Day 3 

21:00 

22:55 

1:55 


Day 4 

21:00 

21:51 

0:51 

Average 




2:18 

Test 8 


Week 1 

Day 1 

21:00 

2:25 

5:25 


Day 2 

21:00 

0:19 

3:19 


Day 3 

21:00 

22:45 

1:45 


Day 4 

21:00 

1:55 

4:55 


Day 5 

21:00 

22:20 

1:20 

Weeks 

Day 1 

21:00 

1:23 

4:23 


Day 2 

21:00 

23:10 

2:10 


Day 3 

21:00 

5:05 

8:05 


Day 4 

21:00 

21:55 

0:55 


Day 5 

21:00 

22:50 

1:50 

Average 




3:24 


Mean 

Time 

between 

Failures 




3:06 


51 




THIS PAGE INTENTIONALLY LEET BLANK 


52 



LIST OF REFERENCES 


Avarez, Kristen, “Fiscal Year 2007 SEAT ASK for DEP Einal,” Technical Report, Naval 
Sea Systems Command (NAVSEA), September 1, 2006. 

Aird, Thomas, Interview with the End-to-End Eaboratory Eead for the Command and 
Control Department, unpublished interview, SPAWAR Systems Center-Pacific (San 
Diego, California), March 16, 2009. 

Battle Eorce Interoperability Management Systems (BFIMS) Web site, 
https://t308 depweb.nswc.navv.mil , September 3, 2009. 

Coyne, Kevin, “The Navy Distributed Engineering Plant (DEP) Strategic Plan for the 
Eiscal Year 2007 and beyond. Rev 1,” Naval Sea Systems Command (NAVSEA), 
August 24, 2006. 

“Distributed Engineering Plant (DEP) Battle Eorce Interoperability Test (BEIT) 
Instructions,” Naval Sea Systems Command (NAVSEA), July 2005. 

“End to End System Engineering (E2E SE) Eab Strategy Vision and Eramework,” Issue 
Brief, SPAWAR Systems Center-Pacific (San Diego, California), May 20, 2008. 

Mahon, John, “Distributed Engineering Plant (DEP) Battle Eorce Interoperability Test 
(BEIT) Semi-Annual Report,” Naval Sea Systems Command (NAVSEA), April 
2003a. 

Mahon, John, “Distributed Engineering Plant (DEP) Battle Eorce Interoperability Test 
(BEIT) Semi-Annual Report,” Naval Sea Systems Command (NAVSEA), October 
2003b. 

Mann, Paul, “Eiscal Year 2005 SEAT ASK for DEP Pinal,” Technical Report, Naval Sea 
Systems Command (NAVSEA), September 29, 2004. 

Miller, Chris, “PEOC4I Governance CONOPS, End-to-End Systems Engineering Team,” 
Space and Naval Warfare Systems Command (SPAWAR), June 02, 2008. 

Monteith, Greg, “The Navy Distributed Engineering Plant (DEP) Overview and History,” 
Issue Brief, Naval Surface Warfare Center Dahlgren, December 2001. 

Musson, Steve, “Strategy for Collaborative Engineering and Certification Environment 
Vision and Eramework,” Issue Brief, SPAWAR Systems Center-Pacific (San Diego, 
California), January 22, 2008a. 

Musson, Steve, “End-to-End System Engineering (E2E SE),” Issue Brief, SPAWAR 
Systems Center-Pacific (San Diego, California), May 20, 2008b. 


53 



Over the Horizon Targeting (OTHT) Web site, https://otht.spawar.navv.mil/ , July 9, 
2008. 

Ragsdale, Cliff T., '^Spreadsheet Modeling & Decision Analysis: A Practical Introduction 
to Management Science, 5e," Thompson South-Western, Ohio, 2007. 

Seaver, Andrian, “Distributed Engineering Plant (DEP) Eorce Interoperability Test (PIT) 
Semi-Annual Report,” Naval Sea Systems Command (NAVSEA), April 2004a. 

Seaver, Andrian, “Distributed Engineering Plant (DEP) Eorce Interoperability Test (PIT) 
Semi-Annual Report,” Naval Sea Systems Command (NAVSEA), December 2004b. 

Tran, Viet, Interview with the DEP Engineering Eead for the Command and Control 
Department, unpublished interview, SPAWAR Systems Center-Pacific (San Diego, 
California), March 19, 2009. 

Wikipedia Web site, http://en.wikipedia.org/wiki/Mean time between failures/ , June 
19, 2009. 

Williams, Mark, Interview with the End-to-End Deputy PM for the Command and 
Control Department, unpublished interview, SPAWAR Systems Center-Pacific (San 
Diego, California), Pebruary 20, 2009. 


54 




INITIAL DISTRIBUTION LIST 


1. Defense Teehnical Information Center 
Ft. Belvoir, Virginia 

2. Dudley Knox Library 
Naval Postgraduate School 
Monterey, California 

3. Dr. Thomas V. Huynh 
Systems Engineering Department 
Naval Postgraduate School 
Monterey, California 

4. Dr. Gary McCown 
SPAWARSYSCEN PACIEIC 
San Diego, California 

5. Thomas Aird 
SPAWARSYSCEN PACIEIC 
San Diego, California 

6. Einh Nguyen 
SPAWARSYSCEN PACIEIC 
San Diego, California 


55 



